Image-processing device and control method thereof

ABSTRACT

An image-processing device is configured to detect a motion region within an input image; generate an intermediate image from an N-th frame and an (N+1)-th frame; perform processing of reducing a high-frequency component of spatial frequency on the intermediate image; and output an image of which a frame rate is increased by insertion of the intermediate image having undergone the processing between the N-th frame and the (N+1)-th frame. When a target position at which a pixel value of the intermediate image is to be calculated lies within the motion region, a pixel value of the intermediate image at the target position is calculated using a pixel value of the N-th frame and a pixel value of the (N+1)-th frame at a position in the vicinity of the target position and outside the motion region.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image-processing device and acontrol method thereof.

2. Description of the Related Art

Conventional frame rate conversion methods in which a frame frequency isincreased by a factor of N (N being a natural number equal to or greaterthan 2) include methods wherein a one-frame image is divided(distributed) into a plurality of sub-frames. Frame rate conversionmethods include methods that involve dividing an input image into asub-frame in which a high-frequency component is pre-emphasized(pre-emphasized sub-frame, hereafter referred to as “Hi image”) and asub-frame in which a high-frequency component is reduced orde-emphasized (de-emphasized sub-frame, hereafter referred to as “Loimage”), and alternately outputting then the sub-frames. Such a methodis called a drive distribution method. This drive distribution methodallows reducing the perception of motion blur that arises in visualtracking (feature of vision of a moving picture whereby a moving objectwithin the image is tracked in the line of sight). Technologies relatingto drive distribution methods are disclosed in, for instance, JapanesePatent Application Publication Nos. 2009-44460 and 2009-42481. Anexplanation follows next on an instance where a drive distributionmethod is implemented, and an instance where it is not, compared witheach other.

FIG. 19A and FIG. 19B are diagrams illustrating an instance where animage identical to an inputted frame image is simply outputted in theform of two sub-frame images, through frame rate conversion in which nodrive distribution method is implemented. FIG. 19A illustrates, as imagedata, the brightness of pixels on a given horizontal line within theimage. The abscissa axis direction represents pixel position, in thehorizontal direction, within the image, and the ordinate axis denotesthe brightness value of each pixel. The square waveform represents thechange in brightness at a given coordinate in the horizontal direction.An input frame is referred to as i frame, a subsequent input frame asi+1 frame, and an intermediate frame between the i frame and the i+1frame is referred to as i+0.5 frame (i: natural number). Movement of thesquare waveform rightward in the horizontal direction indicates thatmotion towards the right in the horizontal direction is taking place inthe image. FIG. 19B illustrates a waveform (image) that is perceived(observed) by an observer when visually tracking the image of FIG. 19Aso as to match the motion of the image. The abscissa axis represents ahorizontal direction coordinate in a coordinate system that conforms tothe motion in visual tracking, and the ordinate axis representsbrightness. The solid line represents the observed waveform, and thedashed line represents an ideal waveform, i.e. what is intended, in theimage data, to be perceived by the observer. The same applies to FIG. 7and FIG. 19A to FIG. 19F.

FIG. 19A illustrates the waveforms of three frames, namely i frame,i+0.5 frame, i+1 frame (referred to as i waveform, i+0.5 waveform andi+1 waveform), for the square waveform that is moving in the horizontalrightward direction. FIG. 19B is the waveform that is observed throughvisual tracking. The waveform actually perceived by human vision uponvisual tracking is a combination of the i waveform and i+0.5 waveform inFIG. 19A, and constitutes a waveform such as the one denoted by thesolid line in FIG. 19B. As can be seen from FIG. 19B, the waveform thatis observed through visual tracking (solid line) has a brightnessportion of intermediate gradation, with respect to the ideal squarewaveform (dashed line). This portion is observed in the form of movingimage blur during visual tracking.

FIG. 19C and FIG. 19D are diagrams illustrating an instance where a Hiimage and a Lo image are alternately outputted through frame rateconversion relying on a drive distribution method. FIG. 19C illustrateswaveforms of a Hi image, a Lo image and a Hi image that are generatedfor the i frame, the i+0.5 frame and the i+1 frame, in accordance with adrive distribution method, for a square waveform that is moving in thehorizontal rightward direction. The waveforms will be referred tohereafter as Hi(i) waveform, Lo(i+0.5) waveform and Hi(i+1) waveform,respectively. FIG. 19D illustrates a waveform (image) that is perceived(observed) by an observer when visually tracking the image of FIG. 19Cso as to match the motion of the image.

The waveform actually perceived by human vision as a result of visualtracking is a combination of the Hi(i) waveform and the Lo(i+0.5)waveform in FIG. 19C, and constitutes a waveform such as the one denotedby the solid line in FIG. 19D. As can be seen from FIG. 19D, the portionat which brightness differs significantly from that of the ideal squarewaveform (dashed line), in the waveform observed through visual tracking(solid line), is small, and there are fewer pixels of intermediategradation as compared with FIG. 19B. As a result, the image is observedwith less moving image blur.

SUMMARY OF THE INVENTION

In the drive distribution method the input image is subjected to afiltering process, to be separated into a component of low spatialfrequency and a component of high spatial frequency. The component oflow spatial frequency is outputted divided into two sub-frames, and thecomponent of high spatial frequency is outputted concentrated in onesub-frame. In the case of frame rate conversion of an image thatincludes a fast-moving object, substantial positional shift occurs inthis method, between sub-frames, at the boundary portion (edge portion)of the object and the background. Accordingly, motion blur may fail tobe sufficiently reduced when tracked. An example of a process of afast-moving image in accordance with a drive distribution method will beexplained with reference to FIG. 19E and FIG. 19F.

FIG. 19E and FIG. 19F are diagrams illustrating an instance where a Hiimage and a Lo image are alternately outputted, through frame rateconversion, in accordance with a drive distribution method, of a squarewaveform moving fast in the horizontal rightward direction. FIG. 19Eillustrates three frames, Hi(i), Lo(i+0.5) and Hi(i+1), generated inaccordance with a conventional drive distribution method. FIG. 19Fillustrates a waveform (image) that is perceived (observed) by anobserver when visually tracking the image of FIG. 19E so as to match themotion of the image. The waveform observed through visual trackingexhibit pixels with a large fall-off in brightness, as denoted by thesolid line in FIG. 19F, with respect to the ideal square waveform(dashed line). That portion is observed as moving image blur.

The present invention suppresses observation of motion blur in a casewhere an observer visually tracks an image having been subjected toframe rate conversion in accordance with a drive distribution method.

A first aspect of the present invention is an image-processing deviceincluding:

a detection unit configured to detect a motion region of motion in animage between an N-th frame image and an (N+1)-th frame image of aninput image;

a generation unit configured to generate an intermediate image from theN-th frame image and the (N+1)-th frame image, on the basis of adetection result by the detection unit;

a processing unit configured to perform on the intermediate image imageprocessing of reducing a high-frequency component of spatial frequency;and an output unit configured to output an image resulting fromincreasing a frame rate of the input image through insertion of theintermediate image having undergone the image processing, between theN-th frame image and the (N+1)-th frame image,

wherein when a target position at which a pixel value of theintermediate image is to be calculated lies within the motion region,the generation unit calculates a pixel value of the intermediate imageat the target position using a pixel value of the N-th frame image and apixel value of the (N+1)-th frame image at a position in the vicinity ofthe target position and outside the motion region.

A second aspect of the present invention is a control method of animage-processing device, the method including:

detecting a motion region of motion in an image between an N-th frameimage and an (N+1)-th frame image of an input image;

generating an intermediate image from the N-th frame image and the(N+1)-th frame image, on the basis of a detection result in thedetecting process;

performing on the intermediate image image processing of reducing ahigh-frequency component of spatial frequency; and

outputting an image resulting from increasing a frame rate of the inputimage through insertion of the intermediate image having undergone theimage processing, between the N-th frame image and the (N+1)-th frameimage,

wherein when in the generating process a target position at which apixel value of the intermediate image is to be calculated lies withinthe motion region, a pixel value of the intermediate image at the targetposition is calculated using a pixel value of the N-th frame image and apixel value of the (N+1)-th frame image at a position in the vicinity ofthe target position and outside the motion region.

The present invention succeeds in suppressing observation of motion blurin a case where an observer visually tracks an image having beensubjected to frame rate conversion in accordance with a drivedistribution method.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments (with reference to theattached drawings).

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of Embodiment 1;

FIG. 2 is a timing chart of Embodiment 1;

FIG. 3A and FIG. 3B are block diagrams of a signal-generating unit 200and an first image-generating unit 300 of Embodiment 1;

FIG. 4 is a flowchart for explaining the operation of the firstimage-generating unit 300 of Embodiment 1;

FIG. 5A to FIG. 5C are diagrams for explaining the operation of thefirst image-generating unit 300 of Embodiments 1 and 2;

FIG. 6 is a block diagram of a distributing unit 400 of Embodiment 1;

FIG. 7A and FIG. 7B are diagrams illustrating, as waveforms, an outputimage and the appearance thereof, in Embodiment 1;

FIG. 8 is a timing chart of Embodiment 2;

FIG. 9A and FIG. 9B are block diagrams of a signal-generating unit 200and an first image-generating unit 300 of Embodiment 2;

FIG. 10A and FIG. 10B are a flowchart for explaining the operation ofthe first image-generating unit 300 of Embodiment 2;

FIG. 11A and FIG. 11B are a flowchart for explaining the operation ofthe first image-generating unit 300 of Embodiment 2;

FIG. 12A and FIG. 12B are block diagrams of an first image-generatingunit 300 of Embodiment 3 and Embodiment 4;

FIG. 13A to FIG. 13E are diagrams illustrating the operation of an firstimage-generating unit 300 and the appearance of an output image inEmbodiment 3;

FIG. 14 is a block diagram of a distributing unit 500 of Embodiment 4;

FIG. 15A and FIG. 15B are diagrams of an first image-generating unit 300of Embodiment 5 and a distributing unit 500 of Embodiment 6;

FIG. 16A to FIG. 16C are diagrams for explaining the operation of thefirst image-generating unit 300 of Embodiment 5;

FIG. 17A and FIG. 17B are diagrams illustrating, as waveforms, an outputimage and the appearance thereof, in Embodiment 5;

FIG. 18A and FIG. 18B are diagrams illustrating differences inprocessing results of an intermediate image depending on a vicinityregion width W, in Embodiment 6; and

FIG. 19A to FIG. 19F are diagrams illustrating instances of visualtracking in conventional art.

DESCRIPTION OF THE EMBODIMENTS Embodiment 1

A first embodiment of the image-processing device and control methodthereof according to the present invention will be explained next. Theimage-processing device in Embodiment 1 increases a frame rate bydividing one frame period of an input image into two sub-frame periods,and alternately outputting, at each sub-frame period, a pre-emphasizedsub-frame (Hi image) and a de-emphasized sub-frame (Lo image) generatedfrom the input image. Embodiment 1 is an example in which the presentinvention is used in an image-processing device in which frame rateconversion is performed in accordance with such a drive distributionmethod. FIG. 1 is a block diagram illustrating the schematicconfiguration of the image-processing device of Embodiment 1. Theimage-processing device of Embodiment 1 illustrated in FIG. 1 has aframe memory 101, a signal-generating unit 200, an firstimage-generating unit 300, a selecting unit 102 and a distributing unit400. The operation of the image-processing device thus configured willbe explained further on. In Embodiment 1, an instance will be explainedwherein an input image having a frame rate of 60 Hz undergoes frame rateconversion to 120 Hz.

FIG. 2 is a timing chart illustrating the operation of Embodiment 1. Inthe figure, N, N+1 and N+2 denote the frame numbers of the input image,such that a larger number translates into a later frame in time. Imagewriting and reading timings in the frame memory 101 will be explainedwith reference to FIG. 2. The input image is written on the frame memory101 at a frame rate of 60 Hz. The image is read from the frame memory101 at a frame rate of 120 Hz, as illustrated in FIG. 2. As illustratedin the figure, image reading from the frame memory 101 is performedthrough two-frame simultaneous reading alternated with one-framereading. That is because two frames are necessary to create the Loimage, and one frame is necessary to create the Hi image.

The image data S1, S2 illustrated in FIG. 1 denote image data that isread alternately from the frame memory 101. The image data S1 is read tocreate the Lo image and the image data S2 is read to create the Hiimage.

<Operation to Output the Lo Image>

The operation at the time of creation of the Lo image will be explainedfirst. An example will be explained of a timing such that the two-frameimage data S1 read from the frame memory 101 is an N frame and an N+1frame.

The signal-generating unit 200 receives the image data S1 and outputs adifference signal HV. The signal-generating unit 200 detects thedirection of motion of a pixel having a small difference of the N frameand the N+1 frame (previous-next frame difference), and of a pixelhaving a large difference. The difference signal HV is a signal of threevalues, as follows.

$\begin{matrix}\begin{matrix}{{HV} = {0\mspace{14mu} \begin{pmatrix}\begin{matrix}{{pixel}\mspace{14mu} {having}\mspace{14mu} a\mspace{14mu} {previous}\text{-}{next}} \\{{frame}\mspace{14mu} {difference}\mspace{14mu} {equal}\mspace{14mu} {to}}\end{matrix} \\{{or}\mspace{14mu} {smaller}\mspace{14mu} {than}\mspace{14mu} a\mspace{14mu} {predefined}\mspace{14mu} {value}}\end{pmatrix}}} \\{= {1\mspace{14mu} \begin{pmatrix}\begin{matrix}{{previous}\text{-}{next}\mspace{14mu} {frame}\mspace{14mu} {difference}} \\{{{larger}\mspace{14mu} {than}\mspace{14mu} {predefined}\mspace{14mu} {value}},}\end{matrix} \\{{horizontal}\mspace{14mu} {motion}\mspace{14mu} {pixel}}\end{pmatrix}}} \\{= {2\mspace{14mu} \begin{pmatrix}\begin{matrix}{{previous}\text{-}{next}\mspace{14mu} {frame}\mspace{14mu} {difference}} \\{{{larger}\mspace{14mu} {than}\mspace{14mu} {predefined}\mspace{14mu} {value}},}\end{matrix} \\{{vertical}\mspace{14mu} {motion}\mspace{14mu} {pixel}}\end{pmatrix}}}\end{matrix} & ( {{Expression}\mspace{14mu} 1} )\end{matrix}$

In Embodiment 1 there are detected two types of motion direction,horizontal and vertical, but the motion directions that are detected arenot limited thereto, and some other direction, or more directions, mayalternatively be detected.

The details of the signal-generating unit 200 will be explained withreference to FIG. 3A. FIG. 3A is a block diagram of thesignal-generating unit 200. The signal-generating unit 200 has adifference-detecting unit 201, a first detecting unit 202, a seconddetecting unit 203, a comparing unit 204 and a determining unit 205. Theimage data of the N frame and the N+1 frame are inputted to thedifference-detecting unit 201, the first detecting unit 202 and thesecond detecting unit 203.

The difference-detecting unit 201 calculates a difference absolute valueof pixel value of a pixel of identical coordinates in the N frame andthe N+1 frame. The difference-detecting unit 201 compares the calculateddifference absolute value with a predefined value set beforehand, andoutputs 1, as a motion signal MV, if the difference absolute value islarger than a predefined value, and 0 if the difference absolute valueis equal to or smaller than a predefined value.

The first detecting unit 202, the second detecting unit 203 and thecomparing unit 204 detect whether the pixel of interest is of horizontalmotion or of vertical motion.

The first detecting unit 202 calculates a difference absolute valuebetween a pixel value N(x, y) of the pixel of interest in the N frameand a pixel value of each pixel within a predefined range, in thehorizontal direction, centered on the pixel of interest, in the N+1frame. The predefined range is set to [x−c, x+c] (c is a constant),where x is the coordinate of the pixel of interest. The first detectingunit 202 calculates respective difference absolute values between thepixel value N(x, y) in the N frame and pixel values N+1 (x−c, y), N+1(x−c+1, y), . . . , N+1 (x+c−1, y), N+1(x+c, y) in the N+1 frame. Thefirst detecting unit 202 outputs a minimum value H from among theplurality of difference absolute values that are calculated. In a casewhere there is a pixel of horizontal motion, the pixel value of thepixel of interest in the N frame and the pixel value of any one pixelwithin the horizontal predefined range in the N+1 frame exhibitordinarily close values, and the minimum value H takes on accordingly asmall value.

The second detecting unit 203 performs the same process as the firstdetecting unit 202, but in the vertical direction, and outputs a minimumvalue V.

The comparing unit 204 outputs 0 as a direction signal D, if the minimumvalue H is equal to or smaller than the minimum value V, and outputs 1if the minimum value H is larger than the minimum value V. In the caseof a pixel with motion in an oblique direction, there is detected thehorizontal direction or vertical direction having the higher correlationwith the direction of oblique motion.

The determining unit 205 generates a difference signal HV on the basisof the motion signal MV and the direction signal D, in accordance withExpression 2 below.

$\begin{matrix}\begin{matrix}{{HV} = {0\mspace{14mu} ( {{{case}\mspace{14mu} {where}\mspace{14mu} {motion}\mspace{14mu} {signal}\mspace{14mu} {MV}} = 0} )}} \\{= {1\mspace{14mu} \begin{pmatrix}{{{case}\mspace{14mu} {where}\mspace{14mu} {motion}\mspace{14mu} {signal}\mspace{14mu} {MV}} = {1\mspace{14mu} {and}}} \\{{{direction}\mspace{14mu} {signal}\mspace{14mu} D} = 0}\end{pmatrix}}} \\{= {2\mspace{14mu} \begin{pmatrix}{{{case}\mspace{14mu} {where}\mspace{14mu} {motion}\mspace{14mu} {signal}\mspace{14mu} {MV}} = {1\mspace{14mu} {and}}} \\{{{direction}\mspace{14mu} {signal}\mspace{14mu} D} = 1}\end{pmatrix}}}\end{matrix} & ( {{Expression}\mspace{14mu} 2} )\end{matrix}$

In Embodiment 1, the motion direction is detected on the basis ofhorizontal difference detection and vertical difference detection, asdescribed above, but motion direction detection is not limited to thatmethod, and for instance some other motion detection scheme by blockmatching may be resorted to. In Embodiment 1, thus, there is detectedfirstly motion within the image on the basis of an N-th frame image andan (N+1)-th frame image of the input image.

The first image-generating unit 300 receives the difference signal HVoutputted by the signal-generating unit 200, and outputs intermediateimage data S3. If a target pixel for which the pixel value in theintermediate image data S3 is calculated is not a pixel within a motionregion, i.e. if the difference signal HV is 0 (pixel of smallprevious-next frame difference), the average value of pixel values ofthe target pixel in the previous and next frames are set as the pixelvalue of the target pixel. If the target pixel is a pixel within themotion region, i.e. if the difference signal HV is other than 0 (pixelof large previous-next frame difference), then a pixel value obtained asa result of the below-described search is used as the pixel value of thetarget pixel.

The details of the first image-generating unit 300 will be explainednext with reference to FIG. 3B. FIG. 3B is a block diagram of the firstimage-generating unit 300. The first image-generating unit 300 has anaveraging unit 301, a horizontal-searching unit 302, avertical-searching unit 303 and a selecting unit 304.

The averaging unit 301 calculates an average value AV of the pixelvalues of a pixel of identical coordinates in the N frame and the N+1frame. The averaging unit 301 outputs the calculated average value AV tothe selecting unit 304.

The horizontal-searching unit 302 sequentially calculates, in order froma pixel that is positioned close to the coordinate of the target pixel,a respective difference absolute value (previous-next frame differenceabsolute value) of the pixel value of the above pixel in the N frame,and the pixel value in the N+1 frame, to search a pixel of smallprevious-next frame difference absolute value. The horizontal-searchingunit 302 calculates, and outputs, a pixel value HP of the target pixel,on the basis of the pixel value of the found pixel. FIG. 4 is a processflow diagram of the horizontal-searching unit 302. In the figures, (x,y) denotes the coordinate value of the target pixel, i denotes a counterfor shifting the coordinate value during the search, and AR denotes apredefined value that delimits the search range.

In step S501, the horizontal-searching unit 302 initializes the counteri to 1.

Steps S502 to 505 constitute a loop. In this loop, thehorizontal-searching unit 302 sequentially calculates, in order from apixel that is positioned close to the coordinate (x, y) of the targetpixel, a respective difference absolute value of the pixel value of theabove pixel in the N frame, and the pixel value in the N+1 frame, tosearch a pixel having a small difference absolute value.

In step S502, the horizontal-searching unit 302 determines whether theprevious-next frame difference absolute value of a search coordinate(x+i, y) is smaller than a predefined value. If the previous-next framedifference absolute value is smaller than a predefined value (S502:Yes), the process proceeds to step S508; if the previous-next framedifference is equal to or greater than the predefined value (S502: No),the process proceeds to step S503.

In step S503, the horizontal-searching unit 302 determines whether theprevious-next frame difference of a search coordinate (x−i, y) issmaller than a predefined value. If the previous-next frame differenceis smaller than a predefined value (S503: Yes), the process proceeds tostep S507; if the previous-next frame difference is equal to or greaterthan the predefined value (S503: No), the process proceeds to step S504.

In step S504, the horizontal-searching unit 302 increments the counteri, and determines in step S505 whether the search of the predefinedrange is completed or not. If the search of the predefined range is notcompleted (S505: No), the process returns to step S502; if the search ofthe predefined range is completed (S505: Yes), the process proceeds tostep S506.

In steps S506, 507 and 508, the horizontal-searching unit 302 calculatesthe pixel value HP of the target pixel in each case.

In step S506, the horizontal-searching unit 302 sets the average valueof the pixel values of the previous and next frames of the target pixelas the pixel value HP of the target pixel.

In steps S507 and 508, the horizontal-searching unit 302 sets, as thepixel value HP of the target pixel, the average value of the pixelvalues of the pixel in the previous and next frames at a position thatis closest, in the horizontal direction, to the target pixel, from amongthe pixels the previous-next frame differences whereof are smaller thanthe predefined value. The horizontal-searching unit 302 outputs thecalculated pixel value HP of the target pixel to the selecting unit 304.

In the above process there is calculated a pixel having a magnitude ofan inter-frame difference, between the pixel value in the N-th frameimage and the pixel value at the (N+1)-th frame image, that is smallerthan a predefined value, from among the pixels within a predefined rangecentered on the target pixel. The average value of the pixel values inthe N-th frame image and the pixel value at the (N+1)-th frame image, ofthe obtained pixel, is set as the pixel value of the target pixel. Inthe above flow there is calculated a pixel at a position that is closestto the target pixel, from among the pixels that have a magnitude ofinter-frame difference, within a predefined range, smaller than thepredefined value. The average value of the pixel values in the N-thframe image and the pixel value at the (N+1)-th frame image, of theobtained pixel, is then set as the pixel value of the target pixel. Whenthere is no pixel having a magnitude of inter-frame difference, withinthe predefined range, that is smaller than the predefined value, thenthe pixel value of the target pixel is set to the average value of thepixel value of the target pixel in the N-th frame image and the pixelvalue at the (N+1)-th frame image. In the above flow, the predefinedrange comprises a range set in the motion direction of a moving object,from the target pixel (horizontal direction rightward orientation), anda range set in an opposite direction (horizontal direction leftwardorientation) of the motion direction, from the target pixel.

The vertical-searching unit 303 performs the same process as thehorizontal-searching unit 302, but in the vertical direction.

The selecting unit 304 selects any one of the average value AV, thepixel value HP and the pixel value VP, in accordance with the differencesignal HV outputted by the signal-generating unit 200, and outputs theselection as the intermediate image data S3. The selecting unit 304selects the average value AV, the pixel value HP or the pixel value VPon the basis of Expression 3 below.

$\begin{matrix}\begin{matrix}{{S\; 3} = {{AV}\mspace{14mu} \begin{pmatrix}{{{difference}\mspace{14mu} {signal}\mspace{14mu} {HV}} = 0} \\\begin{pmatrix}{{case}\mspace{14mu} {where}\mspace{14mu} {previous}\text{-}{next}} \\{{difference}\mspace{14mu} {is}\mspace{14mu} {small}}\end{pmatrix}\end{pmatrix}}} \\{= {{HP}\mspace{14mu} \begin{pmatrix}{{case}\mspace{14mu} {where}\mspace{14mu} {difference}\mspace{14mu} {signal}} \\{{HV} = {1\mspace{14mu} ( {{horizontal}\mspace{14mu} {motion}} )}}\end{pmatrix}}} \\{= {{VP}\mspace{14mu} \begin{pmatrix}{{case}\mspace{14mu} {where}\mspace{14mu} {difference}\mspace{14mu} {signal}} \\{{HV} = {2\mspace{14mu} ( {{vertical}\mspace{14mu} {motion}} )}}\end{pmatrix}}}\end{matrix} & ( {{Expression}\mspace{14mu} 3} )\end{matrix}$

FIG. 5A and FIG. 5B are diagrams for explaining the operation of thefirst image-generating unit 300. FIG. 5A illustrates an image at an Nframe and an N+1 frame. In the N frame and N+1 frame, a white-squareobject moves from the left to the right against a black background, asillustrated in the figure. Herein, Q0 is set as the pixel value of theblack background and Q1 as the pixel value of the object. FIG. 5B is adiagram illustrating the pixel values of pixels of one line denoted byline A in FIG. 5A.

Each pixel in the motion region (x0≦x≦x1) flanked by the coordinates x0,x1 in FIG. 5B yields a difference signal HV=1, and yields a differencesignal HV=0 at a region (x<x0, x1<x) other than the motion region. Asillustrated in FIG. 5B, the motion region is a region between a positionat the boundary between a moving object and the background, in the N-thframe image, and the position of the boundary of the moving object andthe background in the (N+1)-th frame image.

The first image-generating unit 300 sets, as the pixel value of thepixels of difference signal HV=0, the average value of the pixel valueof the pixel in the N frame and the pixel value in the N+1 frame.Accordingly, the pixel value of the pixels of the x<x0 region is Q1, andthe pixel value of the pixels of the x>x1 region is Q0, as denoted bythe intermediate image data S3 of FIG. 5B.

The pixels of difference signal HV=1 take on the pixel value created bythe horizontal-searching unit 302, such that the pixel value of thepixels in the region closer to x0 than the center of the x0≦x≦x1 regionis Q1, and the pixel value of the pixels in the region closer to x1 thanthe center is Q0, as illustrated in FIG. 5B. As a result, the boundarybetween a moving object and the background becomes positioned within themotion region, as illustrated in the graph of the intermediate imagedata S3 of FIG. 5B.

The positional shift of the boundary portion (edge portion) of theobject and the background, between sub-frames, is thus reduced throughinsertion of the generated intermediate image data S3 between the Nframe and the N+1 frame; as a result, it becomes possible to suppressperception of motion blur during visual tracking.

The selecting unit 102 of FIG. 1 alternately switches, at 120 Hz,between the image data S2 read from the frame memory 101 and the imagedata S3 outputted by the first image-generating unit 300, and outputsthe data as the image data S4. Upon output the Lo image, the selectingunit 102 outputs the image data S3 as the image data S4, and upon outputof the Hi image, the selecting unit 102 outputs the image data S2 as theimage data S4.

The distributing unit 400 alternately outputs the Hi image and the Loimage at 120 Hz. FIG. 6 is a block diagram of the distributing unit 400.The distributing unit 400 has an adder 401, an adder 402, a low-passfilter (LPF) 403, a limit unit 404, an adder 405 and a selecting unit406.

The image data S4 is inputted to the adder 401, the adder 402 and theLPF 403. The LPF 403 performs smoothing by applying a spatial LPF to theimage data S4, and outputs the data as the image data S42. The LPF inEmbodiment 1 has five horizontal and five vertical taps, but the numberof taps is not limited thereto. The smoothed image data S42 is outputtedto the adder 402 and the adder 405.

The adder 402 calculates the difference between the image data S42 andthe image data S4 before smoothing, and extracts a high-frequencycomponent that is outputted to the adder 401. The adder 401 adds theimage data S4 and the extracted high-frequency component, and outputsthe result as pre-emphasized image data S41.

The limit unit 404 and the adder 405 perform saturation processing. Anoverview of the limit unit 404 and the adder 405 will be explained next.A known technique can be resorted to herein for saturation processing(Japanese Patent Application Publication No. 2009-44460). Thepre-emphasized image S41 is corrected, in the limit unit 404, in such amanner that pixel values that deviate from a tolerance are kept withinthe tolerance. The tolerance is for instance a range from 0 to 255 ifimage data is 8-bit data. The limit unit 404 outputs the corrected Hiimage to the selecting unit 406, and outputs, to the adder 405, thepixel value fraction departing from the tolerance. The adder 405 addsthe pixel value fraction departing from the tolerance to the image dataS42. The adder 405 outputs the result of the addition, as the Lo image,to the selecting unit 406.

In the distributing unit 400 there is performed image processing ofde-emphasizing the high-frequency component of spatial frequency, andimage processing of pre-emphasizing the high-frequency component ofspatial frequency, for the intermediate image, the N-th frame image andthe (N+1)-th frame image.

Upon display of the Lo image, the selecting unit 406 selects the outputfrom the adder 405. Upon display of the Hi image, the selecting unit 406selects the output from the limit unit 404, and outputs the selection asimage data S5. The image data S5 is displayed on a display panel, notdepicted in the figures.

<Operation to Output a Hi Image>

The operation to output the Hi image will be explained next. An examplewill be explained of a timing such that the image data S2 read from theframe memory 101 is an N+1 frame. Upon output of the Hi image in FIG. 2,the selecting unit 102 selects the image data S2, and outputs this imagedata, as the image data S4, to the distributing unit 400. Thedistributing unit 400 performs the above-described process, creates theHi image and the Lo image, and the selecting unit 406 outputs the Hiimage as the image data S5.

The image-processing device of Embodiment 1 outputs an image having theframe rate of the input image increased through insertion of anintermediate image between the N-th frame image and the (N+1)-th frameimage. Specifically, a one-frame display period is divided into thefirst sub-frame period of display of the Hi image, and the secondsub-frame period of display of the Lo image. In the first sub-frameperiod, image data for one frame is read from the frame memory, theimage data is converted to the Hi image having been pre-emphasized inthe distributing unit 400, and is then outputted. In the secondsub-frame period that follows the first sub-frame period, the image datafor two consecutive frames is read from the frame memory, anintermediate image is generated, the image is converted to the Lo imagehaving been de-emphasized in the distributing unit 400, and is thenoutputted. The image-processing device of Embodiment 1 generates thus,out of images of two temporally consecutive frames, a Lo image of aninstance where the frame rate is increased through display of the Hiimage and the Lo image in one-frame periods. As a result, a Lo image isgenerated in accordance with motion, if any, that is present in theimage.

FIG. 7 is a set of diagrams illustrating an output image in a case ofalternating output of the Hi image and the Lo image in Embodiment 1, andillustrating the appearance of the output image as a waveform. FIG. 7Aillustrates three frames Hi(i), Lo(i+0.5) and Hi(i+1) generated as aresult of the process in Embodiment 1. Visual tracking of such waveformsyields a waveform that is close to an ideal square waveform, asillustrated in FIG. 7B. As a result, the image is observed with lessmoving image blur.

The image-processing device of Embodiment 1 generates the intermediateimage data S3 on the basis of the difference signal HV that is outputtedby the signal-generating unit 200. In the case of difference signalHV=0, the pixel value of each pixel in the intermediate image data S3 isset to the average value of the pixel value of the pixel in the N frameand the pixel value in the N+1 frame.

In the case of difference signal HV=1, there is searched a pixel beingat a position closest to that of the above pixel and that yields adifference signal HV=0 (i.e. a pixel having a previous-next framedifference absolute value smaller than a predefined value). The averagevalue of the pixel value of the found pixel in the N frame and of thepixel value in the N+1 frame is set as the pixel value of the pixel inthe intermediate image data S3. In Embodiment 1, the positional shift ofthe boundary portion (edge portion) between object and backgroundbetween sub-frames is reduced through creation of the Lo image from theintermediate image data and through inter-frame insertion of the Loimage. As a result, it becomes possible to suppress perception of motionblur during visual tracking. Embodiment 1 allows therefore reducingmotion blur during visual tracking of a fast-moving input image, alsoupon frame rate conversion in accordance with a drive distributionmethod.

Embodiment 2

A second embodiment of the image-processing device and control methodthereof according to the present invention will be explained next. InEmbodiment 2 an instance will be explained wherein an input image havinga frame rate of 60 Hz undergoes frame rate conversion to 180 Hz.Embodiment 2 will be explained next with reference to accompanyingdrawings.

Embodiment 2 differs from Embodiment 1 in that the process flow of thesignal-generating unit 200, and of the horizontal-searching unit 302 andthe vertical-searching unit 303 of the first image-generating unit 300are now different from those of Embodiment 1. The process flows of thesignal-generating unit 200, the horizontal-searching unit 302 and thevertical-searching unit 303 will be explained next. Processing unitsother than the signal-generating unit 200 are set to execute processesidentical to those of Embodiment 1. These processing units will bedenoted by the same reference symbols, and a detailed explanationthereof will be omitted.

FIG. 8 is a timing chart illustrating the operation of Embodiment 2. Inthe figure, N, N+1 and N+2 denote the frame numbers of the input image,such that a larger number translates into a newer frame in time. Imagewriting and reading timings in the frame memory 101 will be explainedwith reference to FIG. 8. The input image is written on the frame memory101 at a frame rate of 60 Hz. The image is read from the frame memory101 at a frame rate of 180 Hz, as illustrated in FIG. 8. In Embodiment2, one frame period is divided into three sub-frame periods, such that apre-emphasized sub-frame (Hi image) is displayed at the first sub-frameperiod, and a first and a second de-emphasized sub-frame (Lo_(—)0 image,Lo_(—)1 image) are displayed at a second and a third sub-frame period.

As illustrated in FIG. 8, image reading from the frame memory 101 isperformed by performing, alternately, two-frame simultaneous readingtwice, and one-frame reading once. Image data for two frames isnecessary in order to create the Lo_(—)0 image and the Lo_(—)1 image,while image data for one frame is necessary in order to create the Hiimage. The Lo_(—)0 image is an image of a sub-frame the time-axialposition (temporal position) whereof is closer to the N frame, fromamong the sub-frames for which a de-emphasized sub-frame is displayed.The Lo_(—)1 image is the image of a sub-frame the time-axial positionwhereof is closer to the N+1 frame, from among the sub-frames to becreated. Specifically, the Lo_(—)0 image is a Lo image that is displayedtemporally earlier, from among two Lo images, and the Lo_(—)1 image is aLo image that is displayed temporally later, from among the two Loimages. The image data S1, S2 illustrated in FIG. 1 denote image datathat is read alternately from the frame memory 101. The image data S1 isread to create the Lo_(—)0 image and the Lo_(—)1 image, and the imagedata S2 is read to create the Hi image.

<Operation to Output the Lo Image>

The operation to output the Lo_(—)0 image and the Lo_(—)1 image will beexplained next. An example will be explained of a timing such that thetwo-frame image data S1 read from the frame memory 101 involves an Nframe and an N+1 frame.

The signal-generating unit 200 receives the image data S1 and outputs adifference signal HV. The signal-generating unit 200 detects thedirection of motion of a pixel having a small difference of the N frameand the N+1 frame (previous-next frame difference), and of a pixelhaving a large difference. The difference signal HV is a signal of fivevalues, as follows.

$\begin{matrix}\begin{matrix}{{HV} = {0\mspace{14mu} \begin{pmatrix}\begin{matrix}{{pixel}\mspace{14mu} {having}\mspace{14mu} a\mspace{14mu} {previous}\text{-}{next}} \\{{frame}\mspace{14mu} {difference}\mspace{14mu} {equal}\mspace{14mu} {to}\mspace{14mu} {or}}\end{matrix} \\{{smaller}\mspace{14mu} {than}\mspace{14mu} a\mspace{14mu} {predefined}\mspace{14mu} {value}}\end{pmatrix}}} \\{= {1\mspace{14mu} \begin{pmatrix}{{previous}\text{-}{next}\mspace{14mu} {frame}\mspace{14mu} {difference}\mspace{14mu} {larger}\mspace{14mu} {than}} \\\begin{matrix}{{{predefined}\mspace{14mu} {value}},} \\{{horizontal}\mspace{14mu} {rightward}\mspace{14mu} {motion}\mspace{14mu} {pixel}}\end{matrix}\end{pmatrix}}} \\{= {2\mspace{14mu} \begin{pmatrix}{{previous}\text{-}{next}\mspace{14mu} {frame}\mspace{14mu} {difference}\mspace{14mu} {larger}\mspace{14mu} {than}} \\{{{predefined}\mspace{14mu} {value}},} \\{{horizontal}\mspace{14mu} {leftward}\mspace{14mu} {motion}\mspace{14mu} {pixel}}\end{pmatrix}}} \\{= {3\mspace{14mu} \begin{pmatrix}\begin{matrix}{{previous}\text{-}{next}\mspace{14mu} {frame}\mspace{14mu} {difference}\mspace{14mu} {larger}\mspace{14mu} {than}} \\{{{predefined}\mspace{14mu} {value}},}\end{matrix} \\{{vertical}\mspace{14mu} {upward}\mspace{14mu} {motion}\mspace{14mu} {pixel}}\end{pmatrix}}} \\{= {4\mspace{14mu} \begin{pmatrix}{{previous}\text{-}{next}\mspace{14mu} {frame}\mspace{14mu} {difference}\mspace{14mu} {larger}\mspace{14mu} {than}} \\{{{predefined}\mspace{14mu} {value}},} \\{{vertical}\mspace{14mu} {downward}\mspace{14mu} {motion}\mspace{14mu} {pixel}}\end{pmatrix}}}\end{matrix} & ( {{Expression}\mspace{14mu} 4} )\end{matrix}$

In Embodiment 2 there are detected two types of motion direction,horizontal and vertical, but the motion directions that are detected arenot limited thereto, and some other direction, or more directions, mayalternatively be detected.

The details of the signal-generating unit 200 will be explained withreference to FIG. 9A. FIG. 9A is a block diagram of thesignal-generating unit 200. The signal-generating unit 200 has thedifference-detecting unit 201, a first detecting unit 206, a seconddetecting unit 207, a comparing unit 208 and a determining unit 209. Theimage data of the N frame and the N+1 frame are inputted to thedifference-detecting unit 201, the first detecting unit 206 and thesecond detecting unit 207.

The difference-detecting unit 201 calculates a difference absolute valueof pixel value of a pixel of identical coordinates in the N frame andthe N+1 frame. The difference-detecting unit 201 compares the calculateddifference absolute value with a predefined value set beforehand, andoutputs 1, as the motion signal MV, if the difference absolute value islarger than a predefined value, and 0 if the difference absolute valueis equal to or smaller than a predefined value.

The first detecting unit 206, the second detecting unit 207 and thecomparing unit 208 detect whether the pixel of interest is of horizontalrightward motion, horizontal leftward motion, vertical upward motion orvertical downward motion.

The first detecting unit 206 calculates respective difference absolutevalues between the pixel value N(x, y) of a pixel of interest in the Nframe and the pixel value of each pixel within a predefined range, inthe horizontal direction, centered on the pixel of interest, in the N+1frame. The predefined range is set to [x−c, x+c] (c is a constant),where x is the coordinate of the pixel of interest. The first detectingunit 206 calculates respective difference absolute values between thepixel value N(x, y) in the N frame and pixel values N+1(x−c, y),N+1(x−c+1, y), . . . , N+1(x+c−1, y), N+1(x+c, y) in the N+1 frame. Thefirst detecting unit 206 outputs a minimum value H from among theplurality of difference absolute values that are calculated. The firstdetecting unit 206 outputs 1, as the horizontal direction signal HD, ina case where the motion detected on the basis of the pixel of minimumvalue H is rightward motion, and 0 in the case of leftward motion. In acase where there is a pixel in the horizontal motion, the pixel value ofthe pixel of interest in the N frame and the pixel value of any onepixel within the horizontal predefined range in the N+1 frame exhibitclose values, and the minimum value H takes on accordingly a smallvalue.

The second detecting unit 207 performs the same process as the firstdetecting unit 206, but in the vertical direction, and outputs theminimum value V. The second detecting unit 207 outputs 1, as thevertical direction signal VD, in a case where the motion detected on thebasis of the pixel of minimum value V is upward motion, and 0 in thecase of downward motion.

The comparing unit 208 generates the direction signal D, in accordancewith Expression 5 below, on the basis of the minimum value H, theminimum value V, the horizontal direction signal HD and the verticaldirection signal VD.

$\begin{matrix}\begin{matrix}{D = {0\mspace{14mu} \begin{pmatrix}\begin{matrix}{{case}\mspace{14mu} {where}\mspace{14mu} {minimum}\mspace{14mu} {value}\mspace{14mu} H\mspace{14mu} {is}\mspace{14mu} {equal}\mspace{14mu} {to}\mspace{14mu} {or}} \\{{{smaller}\mspace{14mu} {than}\mspace{14mu} {minimum}\mspace{14mu} {value}\mspace{14mu} V},}\end{matrix} \\{{with}\mspace{14mu} {horizontal}\mspace{14mu} {rightward}\mspace{14mu} {motion}}\end{pmatrix}}} \\{= {1\mspace{14mu} \begin{pmatrix}{{case}\mspace{14mu} {where}\mspace{14mu} {minimum}\mspace{14mu} {value}\mspace{14mu} H\mspace{14mu} {is}\mspace{14mu} {equal}\mspace{14mu} {to}\mspace{14mu} {or}} \\{{{smaller}\mspace{14mu} {than}\mspace{14mu} {minimum}\mspace{14mu} {value}\mspace{14mu} V},} \\{{with}\mspace{14mu} {horizontal}\mspace{14mu} {leftward}\mspace{14mu} {motion}}\end{pmatrix}}} \\{= {2\mspace{14mu} \begin{pmatrix}{{case}\mspace{14mu} {where}\mspace{14mu} {minumum}\mspace{14mu} {value}\mspace{14mu} H\mspace{14mu} {is}\mspace{14mu} {larger}\mspace{14mu} {than}} \\{{{minimum}\mspace{14mu} {value}\mspace{14mu} V},} \\{{with}\mspace{14mu} {vertical}\mspace{14mu} {upward}\mspace{14mu} {motion}}\end{pmatrix}}} \\{= {3\mspace{14mu} \begin{pmatrix}{{case}\mspace{14mu} {where}\mspace{14mu} {minimum}\mspace{14mu} {value}\mspace{14mu} H\mspace{14mu} {is}\mspace{14mu} {larger}\mspace{14mu} {than}} \\{{{minimum}\mspace{14mu} {value}\mspace{14mu} V},} \\{{with}\mspace{14mu} {vertical}\mspace{14mu} {downward}\mspace{14mu} {motion}}\end{pmatrix}}}\end{matrix} & ( {{Expression}\mspace{14mu} 5} )\end{matrix}$

In the case of a pixel with motion in an oblique direction, there isdetected the horizontal direction or vertical direction having thehigher correlation with the direction of oblique motion.

The determining unit 209 generates the difference signal HV on the basisof the motion signal MV and the direction signal D, in accordance withExpression 6 below.

$\begin{matrix}\begin{matrix}{{HV} = {0\mspace{14mu} ( {{{case}\mspace{14mu} {where}\mspace{14mu} {motion}\mspace{14mu} {signal}\mspace{14mu} {MV}} = 0} )}} \\{= {1\mspace{14mu} \begin{pmatrix}{{{case}\mspace{14mu} {where}\mspace{14mu} {motion}\mspace{14mu} {signal}\mspace{14mu} {MV}} = {1\mspace{14mu} {and}}} \\{{{direction}\mspace{14mu} {signal}\mspace{14mu} D} = 0}\end{pmatrix}}} \\{= {2\mspace{14mu} \begin{pmatrix}{{{case}\mspace{14mu} {where}\mspace{14mu} {motion}\mspace{14mu} {signal}\mspace{14mu} {MV}} = {1\mspace{14mu} {and}}} \\{{{direction}\mspace{14mu} {signal}\mspace{14mu} D} = 1}\end{pmatrix}}} \\{= {3\mspace{14mu} \begin{pmatrix}{{{case}\mspace{14mu} {where}\mspace{14mu} {motion}\mspace{14mu} {signal}\mspace{14mu} {MV}} = {1\mspace{14mu} {and}}} \\{{{direction}\mspace{14mu} {signal}\mspace{14mu} D} = 2}\end{pmatrix}}} \\{= {4\mspace{14mu} \begin{pmatrix}{{{case}\mspace{14mu} {where}\mspace{14mu} {motion}\mspace{14mu} {signal}\mspace{14mu} {MV}} = {1\mspace{14mu} {and}}} \\{{{direction}\mspace{14mu} {signal}\mspace{14mu} D} = 3}\end{pmatrix}}}\end{matrix} & ( {{Expression}\mspace{14mu} 6} )\end{matrix}$

In Embodiment 2, the motion direction is detected on the basis ofhorizontal difference detection and vertical difference detection, asdescribed above, but motion direction detection is not limited to thatmethod, and for instance some other motion detection scheme by blockmatching may be resorted to.

The first image-generating unit 300 receives the difference signal HVoutputted by the signal-generating unit 200, and outputs theintermediate image data S3. The pixel value of each pixel in theintermediate image data S3 is set to the average value of the previousand next frames in the case of a pixel the difference signal HV of whichis 0 (pixel of small previous-next frame difference), and is set to apixel value obtained as a result of the below-described search in thecase of a pixel the difference signal HV of which is other than 0 (pixelof large previous-next frame difference). In the search method ofEmbodiment 2, the search pitches in the left-right direction and theup-down direction vary depending on the time-axial positions of thegenerated intermediate images and on the image motion direction. Theratio of search pitch in the left-right direction or the up-downdirection are each set to 1:2 or 2:1 in a case of frame rate conversionwhere the frame rate of the input image is increased three-fold.

The details of the first image-generating unit 300 will be explainednext with reference to FIG. 9B. FIG. 9B is a block diagram of the firstimage-generating unit 300. The first image-generating unit 300 has theaveraging unit 301, a horizontal-searching unit 305, avertical-searching unit 306 and a selecting unit 307.

The averaging unit 301 calculates an average value AV of the pixelvalues of a pixel of identical coordinates in the N frame and the N+1frame. The averaging unit 301 outputs the calculated average value AV tothe selecting unit 307.

The horizontal-searching unit 305 sequentially calculates, in order froma pixel that is positioned close to the coordinate of the pixel ofinterest, a respective difference absolute value (previous-next framedifference absolute value) of the pixel value of the above pixel in theN frame, and the pixel value in the N+1 frame, to search a pixel ofsmall previous-next frame difference absolute value. Thehorizontal-searching unit 305 calculates, and outputs, the pixel valueHP of the pixel of interest, on the basis of the pixel value of thefound pixel. FIG. 10A and FIG. 10B illustrate a process flow diagram ofthe horizontal-searching unit 305 upon creation of the Lo_(—)0 image.FIG. 11A and FIG. 11B illustrate a process flow diagram of thehorizontal-searching unit 305 upon creation of the Lo_(—)1 image. In thefigures, (x, y) denotes the coordinate value of the pixel of interest, idenotes the counter for shifting the coordinate value during the search,and AR denotes the predefined value that delimits the search range.

The process flow of creation of the Lo_(—)0 image will be explained withreference to FIG. 10A and FIG. 10B.

In step S1201, the horizontal-searching unit 305 initializes the counteri to 1.

In step S1202 it is determined whether the pixel of interest is movingrightward or leftward. If the pixel of interest is moving rightward, (ifHV=1) the process proceeds to step S1203.

Step S1203 to step S1206 constitute a loop. In this loop, thehorizontal-searching unit 305 sequentially calculates, in order from apixel that is positioned close to the coordinate (x, y) of the targetpixel, a respective difference absolute value of the pixel value of theabove pixel in the N frame and the pixel value in the N+1 frame, tosearch a pixel having a small difference absolute value.

In step S1203, the horizontal-searching unit 305 determines whether theprevious-next frame difference absolute value of a search coordinate(x+2×i, y) is smaller than a predefined value. If the previous-nextframe difference absolute value is smaller than a predefined value(S1203: Yes), the process proceeds to step S1209, if the previous-nextframe difference absolute value is equal to or greater than thepredefined value (S1203: No), the process proceeds to step S1204.

In step S1204, the horizontal-searching unit 305 determines whether theprevious-next frame difference absolute value of the search coordinate(x−i, y) is smaller than a predefined value. If the previous-next framedifference absolute value is smaller than a predefined value (S1204:Yes), the process proceeds to step S1208, if the previous-next framedifference absolute value is equal to or greater than the predefinedvalue (S1204: No), the process proceeds to step S1205.

In step S1205, the horizontal-searching unit 305 increments the counteri, and determines in step S1206 whether the search of the predefinedrange is completed or not.

In steps S1207, 1208 and 1209, the horizontal-searching unit 305calculates the respective pixel value HP of the pixel of interest ofinstances where the pixel of interest is moving rightward.

In step S1207, the horizontal-searching unit 305 sets the pixel value HPof the pixel of interest to the average value of the pixel values of thepixel of interest in the previous and next frames.

In step S1208, the horizontal-searching unit 305 sets the pixel value HPof the pixel of interest to the average value of the pixel values of thepixel in the previous and next frames having a search coordinate (x−i,y).

In step S1209, the horizontal-searching unit 305 sets the pixel value HPof the pixel of interest to the average value of the pixel values of thepixel in the previous and next frames having a search coordinate (x+2×i,y).

If in step S1202 it is determined that the pixel of interest is movingleftward (HV=2), the process proceeds to step S1210. If HV in step S1202is neither 1 nor 2, the process in the flowchart is terminated.

Steps S1210 to 1213 constitute a loop. In this loop, thehorizontal-searching unit 305 sequentially calculates, in order from apixel that is positioned close to the coordinate (x, y) of the targetpixel, a respective difference absolute value (previous-next framedifference absolute value) of the pixel value of the above pixel in theN frame and the pixel value in the N+1 frame, to search a pixel having asmall difference absolute value.

In step S1210, the horizontal-searching unit 305 determines whether theprevious-next frame difference absolute value of the search coordinate(x+i, y) is smaller than a predefined value. If the previous-next framedifference absolute value is smaller than a predefined value (S1210:Yes), the process proceeds to step S1216; if the previous-next framedifference absolute value is equal to or greater than the predefinedvalue (S1210: No), the process proceeds to step S1211.

In step S1211, the horizontal-searching unit 305 determines whether theprevious-next frame difference absolute value of the search coordinate(x−2×+i, y) is smaller than a predefined value. If the previous-nextframe difference absolute value is smaller than a predefined value(S1211: Yes), the process proceeds to step S1215; if the previous-nextframe difference absolute value is equal to or greater than thepredefined value (S1211: No), the process proceeds to step S1212.

In step S1212, the horizontal-searching unit 305 increments the counteri, and determines in step S1213 whether the search of the predefinedrange is completed or not.

In steps S1214, 1215 and 1216, the horizontal-searching unit 305calculates the respective pixel value HP of the pixel of interest ofinstances where the pixel of interest is moving leftward.

In step S1214, the horizontal-searching unit 305 sets the pixel value HPof the pixel of interest to the average value of the pixel values of thepixel of interest in the previous and next frames.

In step S1215, the horizontal-searching unit 305 sets the pixel value HPof the pixel of interest to the average value of the pixel values, in aprevious and a next frame, of the pixel having a search coordinate(x−2×i, y).

In step S1216, the horizontal-searching unit 305 sets the pixel value HPof the pixel of interest to the average value of the pixel value, in theprevious and next frames, of the pixel having the search coordinate(x+i, y).

The process flow of creation of the Lo_(—)1 image will be explained withreference to FIG. 11A and FIG. 11B.

FIG. 11A and FIG. 11B are process flowcharts of the same flow as in FIG.10A and FIG. 10B. In FIG. 11A and FIG. 11B, the conditional expressionsare different from those of steps S1203, 1204, 1210 and 1211 in FIG. 10Aand FIG. 10B and the calculation expressions are different from those ofsteps S1208, 1209, 1215 and 1216 in FIG. 10A and FIG. 10B. Thehorizontal-searching unit 305 performs a process on the basis of theconditional expressions of steps S1223, 1224, 1230 and 1231 in FIG. 11Aand FIG. 11B. The Lo_(—)1 image is created through calculation of thepixel value HP of a pixel of interest using the calculation expressionssteps S1227, 1228, 1229, 1234, 1235 and 1236.

In the flow of FIG. 10A, FIG. 10B, FIG. 11A and FIG. 11B there is set apredefined range of search, for each intermediate image, in accordancewith the time-axial position of insertion of the intermediate imagebetween the N-th frame image and the (N+1)-th frame image. Thepredefined range comprises a range set in the motion direction of themoving object, from the target pixel (horizontal direction rightwardorientation), and a range set in an opposite direction (horizontaldirection leftward orientation) of the motion direction, from the targetpixel. The closer the time-axial position of insertion of theintermediate image is to the N-th frame image, the wider is thepredefined range in the motion direction. The closer the time-axialposition of insertion of the intermediate image to the (N+1)-th frameimage, the wider is the range in the direction opposite the motiondirection.

If the motion direction is of horizontal rightward orientation in thegeneration of the Lo_(—)0 image close to the N-th frame image (S1202:HV=1), there is searched a wide range from the target pixel, in therightward direction, as denoted by the expressions in S1203 and S1204.If the motion direction is of horizontal leftward orientation (S1202:HV=2), there is searched a wide range from the target pixel, in theleftward direction, as denoted by the expressions in S1210 and S1211.

If the motion direction is of horizontal rightward orientation in thegeneration of the Lo_(—)1 image close to the (N+1)-th frame image(S1222: HV=1), there is searched a wide range from the target pixel, inthe leftward direction, as denoted by the expressions in S1223 andS1224. If the motion direction is of horizontal leftward orientation(S1222: HV=2), there is searched a wide range from the target pixel, inthe rightward direction, as denoted by the expressions in S1230 andS1231.

The vertical-searching unit 306 performs the same process as thehorizontal-searching unit 305, but in the vertical direction, tocalculate the pixel value VP.

The selecting unit 307 selects any one of the average value AV, thepixel value HP and the pixel value VP, in accordance with the differencesignal HV outputted by the signal-generating unit 200, and outputs theselection as the intermediate image data S3. The selecting unit 307selects the average value AV, the pixel value HP or the pixel value VPon the basis of Expression 7 below.

$\begin{matrix}\begin{matrix}{{S\; 3} = {{AV}\mspace{14mu} \begin{pmatrix}{{{difference}\mspace{14mu} {signal}\mspace{14mu} {HV}} = 0} \\( {{case}\mspace{14mu} {where}\mspace{14mu} {previous}\text{-}{next}\mspace{14mu} {difference}\mspace{14mu} {is}\mspace{14mu} {small}} )\end{pmatrix}}} \\{= {{HP}\mspace{14mu} \begin{pmatrix}{{{{case}\mspace{14mu} {where}\mspace{14mu} {difference}\mspace{14mu} {signal}\mspace{14mu} {HV}} = 1},{{or}\mspace{14mu} 2}} \\( {{horizontal}\mspace{14mu} {motion}} )\end{pmatrix}}} \\{= {{VP}\mspace{14mu} \begin{pmatrix}{{{{case}\mspace{14mu} {where}\mspace{14mu} {difference}\mspace{14mu} {signal}\mspace{14mu} {HV}} = 3},{{or}\mspace{14mu} 4}} \\( {{vertical}\mspace{14mu} {motion}} )\end{pmatrix}}}\end{matrix} & ( {{Expression}\mspace{14mu} 7} )\end{matrix}$

FIG. 5C is a diagram for explaining the operation of the firstimage-generating unit 300. FIG. 5A illustrates an image at an N frameand an N+1 frame. In the N frame and N+1 frame, a white-square objectmoves from the left to the right against a black background, asillustrated in the figure. Herein, Q0 is set as the pixel value of theblack background and Q1 as the pixel value of the object. FIG. 5C is adiagram illustrating the pixel value of a pixel at a position denoted byline A in FIG. 5A.

The pixel in the motion region (x0≦x≦x1) flanked by the coordinates x0,x1 in FIG. 5C yields a difference signal HV=1, and yields a differencesignal HV=0 at a region (x<x0, x1<x) other than the motion region. Thefirst image-generating unit 300 sets, as the pixel value of the pixelsof difference signal HV=0, the average value of the pixel value of thepixel in the N frame and the pixel value in the N+1 frame. Accordingly,the pixel value of the pixels of the x<x0 region is Q1, and the pixelvalue of the pixels of the x>x1 region is Q0, as denoted by theintermediate image data S3 of FIG. 5C. The pixels of difference signalHV=1 take on the pixel value created by the horizontal-searching unit305. In the Lo_(—)0 image, the left ⅓ region within the pixel region(x0≦x≦x1) of difference signal HV=1 takes on the pixel value Q1, and theright ⅔ region takes on the pixel value Q0. In the Lo 1 image, the left⅔ region within the pixel region (x0≦x≦x1) of difference signal HV=1takes on the pixel value Q1, and the right ⅓ region takes on the pixelvalue Q0. Thus, the frame rate is increased through insertion, betweenthe N-th frame image and the (N+1)-th frame image, of a plurality ofintermediate images, Lo_(—)0 and Lo_(—)1, having mutually differentpositions of the boundary between the moving object and the background,within the motion region.

The positional shift of the boundary portion (edge portion) of theobject and the background, between sub-frames, is thus reduced throughinsertion of the generated intermediate image data S3 (Lo_(—)0) and S3(Lo_(—)1) between the N frame and the N+1 frame. As a result, it becomespossible to suppress perception of motion blur during visual tracking.

The selecting unit 102 of FIG. 1 alternately switches, at 180 Hz,between the image data S2 read from the frame memory 101 and the imagedata S3 outputted by the first image-generating unit 300, and outputsthe data as the image data S4. Upon display of the Hi image, theselecting unit 102 selects the image data S2, and upon display of theLo_(—)0 image and Lo_(—)1 image, the selecting unit 102 selects theimage data S3. The distributing unit 400 alternately outputs the Hiimage, the Lo_(—)0 image and the Lo_(—)1 image at 180 Hz.

The image-processing device of Embodiment 2 generates the intermediateimage data S3 on the basis of the difference signal HV that is outputtedby the signal-generating unit 200. In the case of difference signalHV=0, the pixel value of each pixel in the intermediate image data S3 isset to the average value of the pixel value of the pixel in the N frameand the pixel value in the N+1 frame. In a case other than signal HV=0,there is searched a pixel being at a position closest to that of theabove pixel and that yields a difference signal HV=0 (i.e. a pixelhaving a previous-next frame difference absolute value smaller than apredefined value). The average value of the pixel value of the foundpixel in the N frame and of the pixel value in the N+1 frame is set asthe pixel value of the pixel in the intermediate image data S3. InEmbodiment 2, this search is performed in the left-right direction orthe up-down direction, in accordance with the motion direction of theimage motion direction, but a weight is further added to the searchpitch, in accordance with the time-axial position of the generatedintermediate image data (depending on whether the position is close tothe N frame, or close to the N+1 frame). The positional shift of theboundary portion (edge portion) between object and background betweensub-frames is reduced through creation, and inter-frame insertion, ofsuch intermediate image data. It becomes accordingly possible tosuppress perception of motion blur during visual tracking. Embodiment 2allows therefore reducing motion blur during visual tracking, also uponframe rate conversion, in accordance with a drive distribution method,of a fast-moving input image.

In Embodiment 2 a method has been described wherein the frame rate ofthe input image is increased three-fold, but Embodiment 2 is not limitedthereto, and may be also in an instance where the frame rate isincreased K-fold (K: natural number). In this case, it is sufficient toappropriately set, in accordance with a frame rate magnifications, thegenerated weight for the time-axial position of the intermediate imageand that is to be added to the search pitch of a case of search of apixel such that the previous-next frame difference absolute valueaccording to the image motion direction at the time of generation of theintermediate image is smaller than a predefined value.

Embodiment 3

A third embodiment of the image-processing device and control methodthereof according to the present invention will be explained next. Theconfiguration of the image-processing device of Embodiment 3 isidentical to that of the image-processing device of Embodiment 1illustrated in FIG. 1. Embodiment 3 differs from Embodiment 1 as regardsthe configuration and function of the first image-generating unit 300.The explanation below will focus on differences with respect toEmbodiment 1. An explanation on the details of the configuration andprocesses in Embodiment 1 will be omitted.

The first image-generating unit 300 receives the difference signal HVoutputted by the signal-generating unit 200, and outputs theintermediate image data S3. In the case of a pixel of difference signalHV being 0 (pixel of small previous-next frame difference), the firstimage-generating unit 300 sets the intermediate image data S3 to theaverage value of the previous and next frames. In the case of a pixel ofdifference signal HV other than 0 (pixel of large previous-next framedifference), the intermediate image data S3 is set to a value calculatedfrom pixel values of previous and next frames derived from abelow-described interpolation calculation.

The details of the first image-generating unit 300 will be explainednext with reference to accompanying drawings. FIG. 12A is a blockdiagram of the first image-generating unit 300. The firstimage-generating unit 300 has a region-detecting unit 3011 and awaveform-generating unit 3021.

The region-detecting unit 3011 detects a region (hereafter referred toas motion region) of motion, in the horizontal direction or the verticaldirection, on the basis of the difference signal HV, and outputs aregion signal AR. In Embodiment 3, the region signal AR takes on thecoordinate values at both ends of the motion region, and any one valuefrom among 0, 1 and 2, as denoted by Expression 8.

$\begin{matrix}\begin{matrix}{{{AR} = 0},0,{0\mspace{14mu} ( {{{case}\mspace{14mu} {of}\mspace{14mu} {difference}\mspace{14mu} {signal}\mspace{14mu} {HV}} = 0} )}} \\{{= {x\text{-}{coordinate}\mspace{14mu} {value}\mspace{14mu} {at}\mspace{14mu} {both}\mspace{14mu} {ends}}},} \\{{{{in}\mspace{14mu} {the}\mspace{14mu} {horizontal}\mspace{14mu} {direction}},{{{of}\mspace{14mu} {the}\mspace{14mu} {motion}\mspace{14mu} {region}};}}} \\{{1\mspace{14mu} ( {{{case}\mspace{14mu} {of}\mspace{14mu} {difference}\mspace{14mu} {signal}\mspace{14mu} {HV}} = 1} )}} \\{{= {y\text{-}{coordinate}\mspace{14mu} {value}\mspace{14mu} {at}\mspace{14mu} {both}\mspace{14mu} {ends}}},} \\{{{{in}\mspace{14mu} {the}\mspace{14mu} {vertical}\mspace{14mu} {direction}},{{{of}\mspace{14mu} {the}\mspace{14mu} {motion}\mspace{14mu} {region}};}}} \\{{2\mspace{14mu} ( {{{case}\mspace{14mu} {of}\mspace{14mu} {difference}\mspace{14mu} {signal}\mspace{14mu} {HV}} = 2} )}}\end{matrix} & ( {{Expression}\mspace{14mu} 8} )\end{matrix}$

Herein, the x-coordinate values at both ends of the motion region, inthe horizontal direction, are the x-coordinate values of pixels, havinga difference signal HV of 1, that are adjacent, in the horizontaldirection, to a pixel having a difference signal HV that is not 1. They-coordinate values at both ends of the motion region, in the verticaldirection, are the y-coordinate values of pixels, having a differencesignal HV of 2, that are adjacent, in the vertical direction, to a pixelhaving a difference signal HV that is not 2.

FIG. 13A to FIG. 13C are diagrams for explaining the operation of thefirst image-generating unit 300. FIG. 13A illustrates an image at an Nframe and an N+1 frame. In the N frame and N+1 frame, a white-squareobject moves from the left to the right against a black background, asillustrated in the figure. Herein, Q0 is set as the pixel value of theblack background and Q1 as the pixel value of the object. FIG. 13B is adiagram illustrating the pixel values of pixels of one line denoted byline A in FIG. 13A. The shaded region in FIG. 13B is a horizontal motionregion of difference signal HV=1. In this case, the region signal AR ofeach pixel within the motion region illustrated in FIG. 13B is set to(x0, x1, 1) by the region-detecting unit 3011.

The waveform-generating unit 3021 generates the intermediate image dataS3 from the region signal AR, the N frame image data and the N+1 frameimage data. In the intermediate image data S3, the waveform-generatingunit 3021 calculates the pixel value of a pixel at the motion region(region of large previous-next frame difference, region where the thirdelement of the region signal AR is =1 or 2) through interpolation ofpixel values of the pixel in a previous and a next frame, according to apredefined interpolation function (cos curve).

FIG. 13C illustrates a waveform of the intermediate image data S3 ascalculated through interpolation of the motion region using a cos curve.The waveform-generating unit 3021 interpolates image data of the motionregion in such a manner that the waveform is continuously connected atboth ends of the motion region, as illustrated in FIG. 13C. Thisinterpolation is carried out on the basis of information on theamplitude and length of the cos curve, and that is held beforehand in atable not shown. In the intermediate image data S3, thewaveform-generating unit 3021 sets the pixel value of a pixel of aregion (region of small previous-next frame difference, region where thethird element of the region signal AR is =0) other than the motionregion, as the average value of the pixel value of the previous and nextframes of the pixel. As a result, there is generated the intermediateimage data S3 such that the pixel value changes continuously from one(Q1), from among Q1 as the pixel value of the pixel of the objectadjacent to the boundary of the object and the background, and Q0 as thepixel value of the pixel of the background, to the other pixel value(Q0), in the motion direction of the object, within the motion region.

In the example illustrated in FIG. 13A to FIG. 13C, an instance has beenexplained of an image where an object is moving the horizontaldirection, but the example is not limited thereto. In the case of animage in which the object is moving in the vertical direction, thereholds difference signal HV=2 and the region signal AR is they-coordinates at both ends, in the vertical direction, of the motionregion. In the intermediate image data S3, as a result, the pixel valueof the pixel at the region of one line in the vertical direction, withinthe range specified by the region signal AR, takes on a value resultingfrom interpolation, using a cos curve, of the pixel values of the pixelin the previous and next frames.

In the example explained in Embodiment 3, a cos curve is used forinterpolation, but the function (waveform) that is used forinterpolation is not limited thereto, and the function may be forinstance a slope waveform that changes linearly, or a step waveform thatchanges abruptly. In this case, an intermediate image is generated suchthat the pixel value changes step-wise from one pixel value (Q1), fromamong Q1 as the pixel value of the pixel of the object adjacent to theboundary of the object and the background, and Q0 as the pixel value ofthe pixel of the background, to the other pixel value (Q0), in themotion direction of the object, within the motion region.

FIG. 13D and FIG. 13E are diagrams illustrating an output image in acase of alternating output of the Hi image and the Lo image inEmbodiment 3, and illustrating the appearance of the output image as awaveform. FIG. 13D illustrates three frames Hi(i), Lo(i+0.5) and Hi(i+1)generated through execution of the process of Embodiment 3 on an imagewith motion towards the right in the horizontal direction. The motionregion detected in Embodiment 3 is deemed to be a region in which theremoves the boundary portion of the object and the background (edgeportion), between frames. In Embodiment 3 there is generated anintermediate image through interpolation, using a cos curve, in such amanner that the image at the edge portion is disposed within this motionregion, in the sub-frames. Through such inter-frame insertion, using theintermediate image as a sub-frame, there is reduced the positionalshift, between sub-frames, of the boundary portion (edge portion) of theobject and the background. Accordingly, it becomes possible to suppressperception of motion blur during visual tracking, as illustrated in FIG.13E.

Embodiment 4

A fourth embodiment of the image-processing device and control methodthereof according to the present invention will be explained next. Acharacterizing feature of Embodiment 4, as compared with Embodiment 3,is that upon output of the Lo image, LPF coefficients are set to bedifferent between a region with motion and a region without motion.Specifically, the degree to which the high-frequency component ofspatial frequency is de-emphasized is set to be different between themotion region and other regions, in intermediate images. Blocks thatoperate similarly to those of Embodiment 3 are denoted by the samereference symbols, and an explanation whereof will be omitted.

FIG. 12B is a diagram illustrating the configuration of animage-processing device of Embodiment 4. In Embodiment 4, the differencesignal HV that is outputted by the signal-generating unit 200 isinputted to a distributing unit 500. The configuration and an operationof the distributing unit 500 are different from those of thedistributing unit 400 in Embodiment 3.

FIG. 14 is a block diagram of the distributing unit 500. Embodiment 4differs from Embodiment 3 in that a coefficient-generating unit 507 isadded to the distributing unit 400. An LPF 503 smoothes the image dataS4 by applying thereto a spatial LPF using coefficients that areinputted through the coefficient-generating unit 507, and outputs theimage data S42.

Upon Lo image output, the coefficient-generating unit 507 generatesdifferent LPF coefficients between a region in which the differencesignal HV is 0 (region without motion), and a region where thedifference signal HV is 1 or 2 (region of horizontal or verticalmotion), and outputs the generated coefficients to the LPF 503. Upon Hiimage output, the coefficient-generating unit 507 generates LPFcoefficients identical to the LPF coefficients that are outputted forthe region where the difference signal HV is 0 at the time of Lo imageoutput, and outputs the generated coefficients to the LPF 503.

In Embodiment 4, the above configuration allows switching the LPFcoefficients, upon output of the Lo image, between different LPFcoefficients for regions with motion and regions without motion. In theLo image, specifically, the region in which the high-frequency componentis de-emphasized through substitution by a cos curve, is subjected to anLPF process of low smoothing effect, while other regions are subjectedto an LPF process of high smoothing effect. As a result, this allowssuppressing excessive smoothing in regions of substitution by a coscurve, in the Lo image, and allows for frame rate conversion of higherquality.

Embodiment 5

A fifth embodiment of the image-processing device and control methodthereof according to the present invention will be explained next. Theconfiguration of the image-processing device of Embodiment 5 isidentical to that of the image-processing device of Embodiment 1illustrated in FIG. 1. Embodiment 5 differs from Embodiment 1 as regardsthe configuration and function of the first image-generating unit 300.The explanation below will focus on differences with respect toEmbodiment 1. An explanation on the details of the configuration andprocesses in Embodiment 1 will be omitted.

The first image-generating unit 300 receives the difference signal HVoutputted by the signal-generating unit 200, and outputs theintermediate image data S3. In the case of a pixel the difference signalHV of which is 0 (pixel of small previous-next frame difference), thefirst image-generating unit 300 sets the intermediate image data S3 tothe average value of the previous and next frames. In the case of apixel the difference signal HV of which is other than 0 (pixel of largeprevious-next frame difference), a value resulting from correcting theaverage value of the previous and next frames is set as the intermediateimage data S3.

The details of the first image-generating unit 300 will be explainednext with reference to accompanying drawings. FIG. 15A is a blockdiagram of the first image-generating unit 300. The firstimage-generating unit 300 has a second image-generating unit 3010, agradation-calculating unit 3020 and a replacing unit 3030.

The second image-generating unit 3010 generates an average image (firstaverage image) in the N frame and the N+1 frame, and outputs the averageimage as an average image AVE.

On the basis of the difference signal HV and the average image AVE, thegradation-calculating unit 3020 calculates an intermediate gradation onthe basis of surrounding pixels, in the horizontal direction or thevertical direction, of each pixel, and outputs the result as anintermediate gradation APL (second average image). Thegradation-calculating unit 3020 calculates the intermediate gradationAPL as described below, in accordance with the value of the differencesignal HV.

$\begin{matrix}\begin{matrix}{{A\; P\; L} = {0\mspace{14mu} ( {{{case}\mspace{14mu} {of}\mspace{14mu} {HV}} = 0} )}} \\{= {{average}\mspace{14mu} {value}\mspace{14mu} {of}\mspace{14mu} {the}\mspace{14mu} {pixel}\mspace{14mu} {values}\mspace{14mu} {in}\mspace{14mu} {the}\mspace{14mu} {AVE}\mspace{14mu} {for}}} \\{{{a\mspace{14mu} {pixel}\mspace{14mu} {within}\mspace{14mu} a\mspace{14mu} {predefined}\mspace{14mu} {range}\mspace{14mu} ( {{first}\mspace{14mu} {range}} )},}} \\{{{{in}\mspace{14mu} {the}\mspace{14mu} {horizontal}\mspace{14mu} {direction}},}} \\{{{centered}\mspace{14mu} {on}\mspace{14mu} {the}\mspace{14mu} {pixel}\mspace{14mu} {of}\mspace{14mu} {interest}\mspace{14mu} ( {{{case}\mspace{14mu} {of}\mspace{14mu} {HV}} = 1} )}} \\{= {{average}\mspace{14mu} {value}\mspace{14mu} {of}\mspace{14mu} {the}\mspace{14mu} {pixel}\mspace{14mu} {values}\mspace{14mu} {in}\mspace{14mu} {the}\mspace{14mu} {AVE}\mspace{14mu} {for}}} \\{{a\mspace{14mu} {pixel}\mspace{14mu} {within}\mspace{14mu} a\mspace{14mu} {predefined}\mspace{14mu} {range}\mspace{14mu} ( {{first}\mspace{14mu} {range}} )}} \\{{{{in}\mspace{14mu} {the}\mspace{14mu} {vertical}\mspace{14mu} {direction}},}} \\{{{centered}\mspace{14mu} {on}\mspace{14mu} {the}\mspace{14mu} {pixel}\mspace{14mu} {of}\mspace{14mu} {interest}\mspace{14mu} ( {{{case}\mspace{14mu} {of}\mspace{14mu} {HV}} = 2} )}}\end{matrix} & ( {{Expression}\mspace{14mu} 9} )\end{matrix}$

In Embodiment 5, the average value of the pixel values in the AVE for apixel within a predefined range is set thus as the intermediategradation, but the method for calculating the intermediate gradation isnot limited thereto, and the intermediate gradation may be calculatedusing a median filter or the like.

The replacing unit 3030 corrects the average image AVE on the basis ofthe average image AVE, the intermediate gradation APL and the differencesignal HV, and outputs the result as the intermediate image data S3.

In the case of difference signal HV=0 for the pixel of interest, thereplacing unit 3030 outputs, as-is, the pixel value of the pixel ofinterest in the average image AVE, as the intermediate image data S3 ofthe pixel of interest.

The replacing unit 3030 performs the process below in the case ofdifference signal HV=1 for the pixel of interest (case where the pixelof interest is a pixel moving in the horizontal direction). Thereplacing unit 3030 calculates a difference absolute value between thepixel value of the pixel of interest for intermediate gradation APL andthe pixel value of a pixel within a predefined range (second range), inthe horizontal direction, centered on the pixel of interest of theaverage image AVE, for each pixel within the predefined range. Herein,the predefined range is set to [x−w, x+w] (w is a constant), where x isthe coordinate of the pixel of interest. Specifically, the replacingunit 3030 calculates respective difference absolute values betweenAPL(x, y), and AVE(x−w, y), AVE(x−w+1, y), . . . , AVE(x+w−1, y) andAVE(x+w, y) (w is a constant). The pixel value having the greatestdifference absolute value with respect to the pixel value of the pixelof interest for the intermediate gradation APL, from among the pixelvalues of pixels within the predefined range of the average image AVE,is set as the pixel value in the intermediate image data S3 of the pixelof interest. That is, the replacing unit 3030 replaces the pixel ofinterest in the average image AVE of the motion region of HV=1 by apixel in the average image AVE within a predefined range from the pixelof interest and such that the difference with respect to the APL islarge. The intermediate image data S3 generated as a result of suchreplacement process is compared with the average image AVE, to narrowthe range of intermediate pixel values in the motion region.

The replacing unit 3030 performs a process identical to that of thedifference signal HV=1, in the vertical direction, for a pixel havingdifference signal HV=2 (pixel with motion in the vertical direction),and outputs the result as the intermediate image data S3.

FIG. 16A to FIG. 16C are diagrams for explaining the operation of thefirst image-generating unit 300. FIG. 16A illustrates an image at an Nframe and an N+1 frame. In the N frame and N+1 frame, a white-squareobject moves from the left to the right against a black background, asillustrated in the figure. Herein, Q0 is set as the pixel value of theblack background and Q1 as the pixel value of the object. FIG. 16B is adiagram illustrating the pixel values of pixels of one line denoted byline A in FIG. 16A. The shaded region in FIG. 16B is a horizontal motionregion of difference signal HV=1.

FIG. 16C illustrates the average image AVE, the intermediate gradationAPL and intermediate image data S3.

In the average image AVE of the N frame and N+1 frame as outputted bythe second image-generating unit 3010, the pixel value for x<x0 is Q1,the pixel value for x0≦x≦x1 is the average value of Q1 and Q0, and thepixel value for x>x1 is Q0, as illustrated in FIG. 16C.

In the intermediate gradation APL outputted by the gradation-calculatingunit 3020, the pixel value at the regions (x<x0, x1<x) of differencesignal HV=0 is 0, as illustrated in FIG. 16C. The pixel value at theregion (x0≦x≦x1) of difference signal HV=1 is the average value of thepixel values of the average image AVE within the predefined range in thehorizontal direction centered on the pixel of interest, and takes on avalue that decreases gradually, towards a small value, from a value thatis larger than the average value of Q1 and Q0.

In the intermediate image data S3 outputted by the replacing unit 3030,the pixel value at the regions (x<x0, x1<x) of difference signal HV=0takes on a value identical to that in the average image AVE, asillustrated in FIG. 16C. The pixel value at the region (x0≦x≦x1) ofdifference signal HV=1 takes on an pixel value of the average image APLfor which the difference absolute value with respect to the pixel valueof intermediate gradation APL of the pixel of interest is greatestwithin a predefined range in the horizontal direction. The regiondenoted by the arrow APL(xa) in the figure is a predefined range in thehorizontal direction (range of ±w centered on the pixel of interest). Inthe region near x0, the pixel value of the average image AVE for whichthe difference absolute value with respect to the intermediate gradationAPL is greatest is Q1, and hence the pixel value of the intermediateimage data S3 is Q1. In the region near x1, the pixel value of theaverage image AVE for which the difference absolute value with respectto the intermediate gradation APL is greatest is Q0, and hence the pixelvalue of the intermediate image data S3 is Q0. If the pixel of interestis a pixel in the vicinity of the middle between x0 and x1, thepredefined range (x±w) centered on the pixel of interest is set to beencompassed by the range x0 to x1. In this case, the pixel value of theaverage image AVE for which the difference absolute value with respectto the intermediate gradation APL is greatest is the average value of Q1and Q0, and hence the pixel value of the intermediate image data S3takes on the average value of Q1 and Q0. As a result, an image thatchanges into the background image, from the object image in the vicinityof the middle of the region of difference signal HV=1, is outputted asthe intermediate image data S3, as illustrated in FIG. 16C. Asillustrated in FIG. 16C, a comparison between the average image AVE andthe intermediate image data S3 reveals that the region of intermediatepixel value between the pixel values Q1 and Q0 lies in the range x0 tox1, for the average image AVE, but lies within a narrower range for theintermediate image data S3. Thus, an intermediate image is generatedsuch that the pixel value changes step-wise from one pixel value (Q1),from among Q1 as the pixel value of the pixel of the object adjacent tothe boundary of the object and the background, and Q0 as the pixel valueof the pixel of the background, to the other pixel value (Q0), in themotion direction of the object, within the motion region.

In the example illustrated in FIG. 16A to FIG. 16C an instance has beenexplained of an image where an object is moving the horizontaldirection, but the example is not limited thereto. In the case of animage in which an image is moving in the vertical direction, there holdsdifference signal HV=2, and there is calculated an intermediategradation from the surrounding pixels in the vertical direction. Thereplacing unit 3030 calculates a pixel value of an average image suchthat the difference absolute value with respect to the intermediategradation is greatest, from pixels in a predefined range (range of y±w)in the vertical direction, and outputs the pixel value of the averageimage as the intermediate image.

FIG. 17A and FIG. 17B are diagrams illustrating an output image in acase of alternating output of the Hi image and the Lo image inEmbodiment 5, and illustrating the appearance of the output image in theform of a waveform. FIG. 17A illustrates three frames Hi(i), Lo(i+0.5)and Hi(i+1) generated through execution of the process of Embodiment 5,on an image with motion towards the right in the horizontal direction.In the process of Embodiment 5 there is narrowed the region of theintermediate gradation that is generated through calculation of theaverage of the N frame and the N+1 frame. In the intermediate image dataS3, as a result, the image at the boundary portion (edge portion) of theobject and the background is disposed close to the center of the motionregion. The positional shift of edge portions between sub-frames becomessmaller as a result, which makes it possible to suppress perception ofmotion blur during visual tracking, as illustrated in FIG. 17B.

Embodiment 6

A sixth embodiment of the image-processing device and control methodthereof according to the present invention will be explained next. Thecharacterizing feature of Embodiment 6, as compared with Embodiment 5,is that now the width of a vicinity region for reference duringcorrection is modified in accordance with the speed of motion of theinput image. Blocks that operate similarly to those of Embodiment 5 aredenoted by the same reference symbols, and an explanation whereof willbe omitted. The internal configuration and operation of the firstimage-generating unit 300 in Embodiment 6 are different from those ofthe first image-generating unit 300 in Embodiment 5.

FIG. 15B is a block diagram of the first image-generating unit 300. Thefirst image-generating unit 300 in Embodiment 6 differs from that ofEmbodiment 5 illustrated in FIG. 15A in that now the firstimage-generating unit 300 has additionally a width-calculating unit 502.

The replacing unit 501 performs a correction process on the averageimage AVE, using a vicinity region width W that is inputted from thewidth-calculating unit 502.

The replacing unit 501 performs a process as described below, inaccordance with the difference signal HV.

In the case of difference signal HV=0 for the pixel of interest, thereplacing unit 501 outputs, as-is, the pixel value of the pixel ofinterest in the average image AVE, as the intermediate image data S3 ofthe pixel of interest.

The replacing unit 501 performs the process below in the case ofdifference signal HV=1 for the pixel of interest (case where the pixelof interest is a pixel moving in the horizontal direction). Thereplacing unit 501 calculates a difference absolute value between thepixel value of intermediate gradation of the pixel of interest and thepixel value in the average image AVE of a pixel within a range ofvicinity region width W centered on the pixel of interest, in thehorizontal direction, for each pixel within the above range. Herein, therange of vicinity region width W is [x−W, x+W] (W is acquired from thewidth-calculating unit 502), where x is the coordinate of the pixel ofinterest. The replacing unit 501 calculates respective differenceabsolute values between APL(x, y) and AVE(x−W, y), AVE(x−W+1, y), . . ., AVE(x+W−1, y) and AVE(x+W, y). The pixel value of the average imageAVE at the position at which the difference absolute value takes on amaximum value is outputted as the intermediate image data S3.

The replacing unit 501 performs a process identical to that in the caseof a difference signal HV=1, in the vertical direction, for a pixelhaving difference signal HV=2 (pixel with motion in the verticaldirection), and outputs the result as the intermediate image data S3.

The width-calculating unit 502 calculates a motion width, in thehorizontal direction or vertical direction, from the difference signalHV, calculates the vicinity region width W from the motion width, andoutputs the result. In Embodiment 6 the vicinity region width W iscalculated on the basis of Expression 10 below.

$\begin{matrix}\begin{matrix}{W = {0\mspace{14mu} ( {{{case}\mspace{14mu} {of}\mspace{14mu} {difference}\mspace{14mu} {signal}\mspace{14mu} {HV}} = 0} )}} \\{= {{motion}\mspace{14mu} {{width}/2}\mspace{14mu} {in}\mspace{14mu} {the}\mspace{14mu} {horizontal}\mspace{14mu} {direction}}} \\{( {{{case}\mspace{14mu} {of}\mspace{14mu} {difference}\mspace{14mu} {signal}\mspace{14mu} {HV}} = 1} )} \\{= {{motion}\mspace{14mu} {{width}/2}\mspace{14mu} {in}\mspace{14mu} {the}\mspace{14mu} {horizontal}\mspace{14mu} {direction}}} \\{( {{{case}\mspace{14mu} {of}\mspace{14mu} {difference}\mspace{14mu} {signal}\mspace{14mu} {HV}} = 1} )}\end{matrix} & ( {{Expression}\mspace{14mu} 10} )\end{matrix}$

The motion width in the horizontal direction is calculated as follows.Firstly, there is calculated the x-coordinate values of both ends of themotion region in the horizontal direction. Herein, the x-coordinatevalues at both ends of the motion region, in the horizontal direction,are the x-coordinate values of pixels having a difference signal HV of 1and being adjacent, in the horizontal direction, to a pixel having adifference signal HV that is not 1. A value resulting from adding 1 tothe difference absolute value of the x-coordinate values of both ends inthe horizontal direction thus calculated is set as the motion width inthe horizontal direction. The motion width in the vertical direction iscalculated in the same way as for the horizontal direction.

As a result, it becomes possible to correct the average image AVE of afast-moving image by referring to a wide vicinity region, and to correctthe average image AVE of a slow-moving image by referring to a narrowvicinity region. Embodiment 6 allows setting the vicinity region width Wto an appropriate value in accordance with the image motion speed.

FIG. 18A and FIG. 18B are diagrams illustrating the intermediate imagedata S3 in a case where image motion speed is corrected using anexcessively large vicinity region width W, or using an appropriatevicinity region width W is used.

FIG. 18A is a diagram illustrating an instance where image motion speedis corrected using an excessively large vicinity region width W. Asillustrated in FIG. 18A, the motion region of the intermediate imagedata S3 exhibits a waveform in which brightness fluctuatessignificantly. This arises from the fact that the pixel value of theaverage image AVE for which the difference absolute value with respectto the intermediate gradation APL(xa) of the pixel of interest isgreatest is not AVE(x0), but AVE(x1), because the range of the averageimage AVE for calculation of the difference absolute value with respectto the intermediate gradation APL is excessively wide.

FIG. 18B, by contrast, is a diagram illustrating an instance where imagemotion speed is corrected using an appropriate vicinity region width W.As illustrated in FIG. 18B, the region of intermediate gradation becomesnarrower and an image of the boundary portion (edge portion) of theobject and the background is disposed close to the center of the motionregion of the intermediate image data S3, thanks to the use of anappropriate vicinity region width W. This arises from the fact that thepixel value of the average image AVE for which the difference absolutevalue with respect to the intermediate gradation APL(xa) of the pixel ofinterest is greatest is AVE(x0), because the range of the average imageAVE for calculation of the difference absolute value with respect to theintermediate gradation APL is appropriate.

Through calculation of an appropriate vicinity region width W inaccordance with image motion speed, it becomes thus possible inEmbodiment 6 to narrow properly the region of intermediate gradation,also in a case of fast image motion, and to suppress significantfluctuation of brightness of the intermediate image at a motion region,also in the case of slow image motion. Accordingly, motion blur can bereduced upon visual tracking, regardless of the image motion speed.

Other Embodiments

The present invention can be carried out also by a computer (or devicesuch as a CPU or MPU) of a device or system that realizes the functionsof the above-described embodiments through reading and execution of aprogram that is recorded on a storage device. Furthermore, for instance,the present invention can be carried out also in accordance with amethod that comprises steps that are executed by a computer of a deviceor system that realizes the functions of the above-described embodimentsthrough reading and execution of a program that is recorded on a storagedevice. To that end, the program may be supplied to the computer forinstance via a network, or out of various types of recording media thatcan constitute the above storage device (i.e. a non-transitorycomputer-readable storage medium). Accordingly, the scope of the presentinvention includes any and all of the above computer (including devicessuch as a CPU, a MPU and the like), method, program (including programcodes and program products) and computer-readable recording media inwhich the program is held non-transitorily.

Other Embodiments

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2013-245280, filed on Nov. 27, 2013, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image-processing device, comprising: adetection unit configured to detect a motion region in which there ismotion between an N-th frame image and an (N+1)-th frame image of aninput image; a generation unit configured to generate an intermediateimage from the N-th frame image and the (N+1)-th frame image, on thebasis of a detection result by the detection unit; a processing unitconfigured to perform on the intermediate image image processing ofreducing a high-frequency component of spatial frequency; and an outputunit configured to output an image resulting from increasing a framerate of the input image through insertion of the intermediate imagehaving undergone the image processing, between the N-th frame image andthe (N+1)-th frame image, wherein when a target position at which apixel value of the intermediate image is to be calculated lies withinthe motion region, the generation unit calculates a pixel value of theintermediate image at the target position, using a pixel value of theN-th frame image and a pixel value of the (N+1)-th frame image at aposition in the vicinity of the target position and outside the motionregion.
 2. The image-processing device according to claim 1, wherein thedetection unit detects, as a motion region, a region at which themagnitude of an inter-frame difference, which is the difference betweenthe pixel value of the N-th frame image and the pixel value of the(N+1)-th frame image, is equal to or greater than a predefined value. 3.The image-processing device according to claim 1, wherein when thetarget position is within the motion region, the generation unitcalculates a pixel value of the intermediate image at the targetposition on the basis of an average value of the pixel value of the N-thframe image and the pixel value of the (N+1)-th frame image at aposition that is within a predefined range centered on the targetposition and outside the motion region.
 4. The image-processing deviceaccording to claim 1, wherein when the target position is within themotion region, the generation unit calculates a pixel value of theintermediate image at the target position on the basis of an averagevalue of the pixel value of the N-th frame image and the pixel value ofthe (N+1)-th frame image at a position that is within a predefined rangecentered on the target position, outside the motion region, and that isclosest to the target position.
 5. The image-processing device accordingto claim 3, wherein when there is no position that is within thepredefined range centered on the target position and outside the motionregion, the generation unit calculates a pixel value of the intermediateimage at the target position on the basis of an average value of thepixel value of the N-th frame image and the pixel value of the (N+1)-thframe image at the target position.
 6. The image-processing deviceaccording to claim 3, wherein the predefined range comprises a rangethat is set, from the target position, in a direction of motion asdetected by the detection unit, and a range that is set, from the targetposition, in a direction opposite the direction of motion as detected bythe detection unit.
 7. The image-processing device according to claim 1,wherein when the target position is outside the motion region, thegeneration unit calculates a pixel value of the intermediate image atthe target position on the basis of an average value of the pixel valueof the N-th frame image and the pixel value of the (N+1)-th frame imageat the target position.
 8. The image-processing device according toclaim 3, wherein the generation unit generates a plurality ofintermediate images including a first intermediate image and a secondintermediate image, the generation unit using, in order to calculate apixel value of each pixel of the first intermediate image, pixel valuesof the N-th frame image and the (N+1)-th frame image at a firstposition, the generation unit using, in order to calculate a pixel valueof each pixel of the second intermediate image, pixel values of the N-thframe image and the (N+1)-th frame image at a second position, the firstposition and the second position being different from each other.
 9. Theimage-processing device according to claim 8, wherein the generationunit sets the predefined range for each intermediate image in accordancewith a time-axial position, between the N-th frame image and the(N+1)-th frame image, at which the intermediate image is inserted. 10.The image-processing device according to claim 9, wherein the predefinedrange comprises a range that is set, from the target position, in adirection of motion as detected by the detection unit, and a range thatis set, from the target position, in a direction opposite the directionof motion as detected by the detection unit, such that the closer thetime-axial position of insertion of the intermediate image is to theN-th frame image, the wider the range is in the direction of the motion,and the closer the time-axial position of insertion of the intermediateimage is to the (N+1)-th frame image, the wider the range is in thedirection opposite the direction of the motion.
 11. The image-processingdevice according to claim 1, wherein the generation unit generates anintermediate image in which a pixel value varies continuously within themotion region, using a predefined interpolation function.
 12. Theimage-processing device according to claim 11, wherein the processingunit causes a degree at which the high-frequency component of spatialfrequency is reduced to be different between the motion region and otherregions, in the intermediate image.
 13. The image-processing deviceaccording to claim 1, wherein the generation unit generates anintermediate image in which a pixel value varies step-wise within themotion region.
 14. The image-processing device according to claim 13,wherein the generation unit further generates a first average image anda second average image from the N-th frame image and the (N+1)-th frameimage; calculates a pixel value of the first average image at a targetposition, at which a pixel value of the first average image is to becalculated, on the basis of an average value of the pixel value of theN-th frame image at the target position and the pixel value of the(N+1)-th frame image at the target position; calculates a pixel value ofthe second average image at a target position, at which a pixel value ofthe second average image is to be calculated, on the basis of an averagevalue of pixel values of the first average image within a first rangecentered on the target position, when the target position is within themotion region; sets, as a pixel value of the intermediate image at atarget position, at which a pixel value of the intermediate image is tobe calculated, a pixel value of the first average image at a position,from among positions within a second range centered on the targetposition, the magnitude of the difference between a pixel value of thefirst average image at the position and a pixel value of the secondaverage image at the position being greatest, when the target positionis within the motion region.
 15. The image-processing device accordingto claim 14, wherein the first range and the second range comprise arange that is set, from the target position, in a direction of motion asdetected by the detection unit, and a range that is set, from the targetposition, in a direction opposite the direction of motion as detected bythe detection unit;
 16. The image-processing device according to claim14, wherein the second range is set in accordance with a speed of motionas detected by the detection unit.
 17. The image-processing deviceaccording to claim 16, wherein the second range is set to be wider thefaster the speed of motion as detected by the detection unit is.
 18. Theimage-processing device according to claim 1, wherein the processingunit further performs image processing of emphasizing the high-frequencycomponent of spatial frequency on the N-th frame image and the (N+1)-thframe image; and the output unit outputs an image resulting fromincreasing the frame rate of the input image through insertion of theintermediate image having undergone the image processing, between theN-th frame image and the (N+1)-th frame image having undergone the imageprocessing.
 19. A control method of an image-processing device, themethod comprising: detecting a motion region in which there is motionbetween an N-th frame image and an (N+1)-th frame image of an inputimage; generating an intermediate image from the N-th frame image andthe (N+1)-th frame image, on the basis of a detection result in thedetecting process; performing on the intermediate image image processingof reducing a high-frequency component of spatial frequency; andoutputting an image resulting from increasing a frame rate of the inputimage through insertion of the intermediate image having undergone theimage processing, between the N-th frame image and the (N+1)-th frameimage, wherein, in the generating process, when a target position atwhich a pixel value of the intermediate image is to be calculated lieswithin the motion region, a pixel value of the intermediate image at thetarget position is calculated using a pixel value of the N-th frameimage and a pixel value of the (N+1)-th frame image at a position in thevicinity of the target position and outside the motion region.